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DETAILED ACTION 

A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1.17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.114. Applicant's submission filed on 08 January 2007 has been entered. 



Response to Arguments 

Applicant's arguments with respect to claims 46, 48, 50 - 54, 56, and 58 - 60 have been 
considered but are moot in view of the new ground(s) of rejection. 

Applicant's arguments with respect to claims 61 and 62, filed 08 January 2007, have been 
fully considered but they are not persuasive. 

Applicant argues that Harrington nor Rhoads teaches a 3-D animated virtual model 
(pages 8 and 9). Rhoads discloses embedding watermark in animated 3D graphical objects and 
the embedded object can be composited with a video stream to form a video program, such as a 
movie or television program and graphical objects that link to information or electronic 
commerce transactions can be added to a video product, such as a movie, when its converted 
from one format to another (column 18, lines 5-18). Rhoads is brought in to teach animated 3D 
graphical objects as he discloses animated 3D graphical objects. Embedding a watermark into an 
image would not teach the limitation of a 3-D animated virtual model. 

Applicant argues that Harrington does not teach or disclose a need for displaying a 3D 
object in different poses to highlight different perspectives of the 3D object (page 9). The 
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claimed limitation is not expressly recited in claims 61 and 62. The claim limitation recites, 
"producing the augmented reality video by rendering the three-dimensional image data model of 
the product superimposed on the moving model plane in the video data, wherein a rendering of 
the three dimensional image data model has substantially the pose of the moving model plane". 
Harrington discloses the unprojection step comprises the mathematical calculation comprising 
taking a two-dimensional image and determining what three-dimensional object could have 
created that two-dimensional image with respect to what the object can be and where it could 
have been positioned and the subject invention utilizes known dimensions with regard to the 
camera system and viewing plane, in combination with assumed characteristics of the piece of 
paper (column 7, lines 28 - 53). In other words, a three-dimensional object is rendered in the 
computer and the calculations are performed to transpose the three-dimensional object to fit on 
the piece of paper, which superimposes a two-dimensional image. The three-dimensional object 
being transposed to fit on a piece of paper, producing a two-dimensional image, reads on the 
mentioned limitation of the claim. The pose of the two-dimensional image takes the orientation 
of the piece of paper. 

Claim Objections 

Claim 54 is objected to because of the following informalities: claim 54 cites, "the 
method, the method steps comprising:" at the end of the preamble. Appropriate correction is 
required. 
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Claim Rejections - 35 USC § 101 
35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or 
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

Claims 46, 48, 50 - 53, 61, and 62 are rejected under 35 U.S.C. 101 because the claimed 
invention is directed to non-statutory subject matter. Claims 46, 48, 50 - 54, 56, and 58 - 62 
appear to be directed to an abstract idea rather than a practical application of the idea. Practical 
application produces a useful, tangible and concrete result. In determining whether the claim is 
for a "practical application," the focus is not on whether the steps taken to achieve a particular 
result are useful, tangible and concrete, but rather that the final result achieved by the claimed 
invention is "useful, tangible and concrete." The claimed invention does not resulting a physical 
transformation nor does the claimed invention appear to provide a useful, concrete and tangible 
result. Specifically, the claimed invention does not appear to produce a tangible result because 
merely superimposing the rendered 3D model of the product on the moving model plane in the 
video data is nothing more than thoughts or computations within a processor. It fails to use or 
make available for use of the final result. 

Claims 54, 56, and 58 - 60 are rejected under 35 U.S.C. 101 because the claimed 
invention is directed towards nonfunctional descriptive material. Claims 54, 56, and 58 - 60 are 
as such do not exclude nonfunctional descriptive material. A computer-readable medium 
encoded with a computer program is patent eligible subject matter if it is a computer element, 
which defines structure and function interrelationships between the computer program and the 
computer components, which permits the computer program's functionality to be realized. The 
claims are directed to nonfunctional descriptive material and hence are nonstatutory. 
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Claim Rejections - 35 USC §102 
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed publication in this 
or a foreign country, before the invention thereof by the applicant for a patent 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

Claims 46, 52 - 54, and 60 are rejected under 35 U.S.C. 102(a) as being anticipated by 
Kato et al. ("Marker Tracking and HMD Calibration for a Video-based Augmented Reality 
Conferencing System", 20 October 1999). 

Regarding independent claim 46, Kato discloses an image with a whiteboard and ID 
cards that has markers (page 3, column 1, line 9 - column 2, line 4 and Figure 4), which reads on 
the claimed providing video data comprising images of a moving model plane having markers. 
Kato discloses calibrating the Head Mounted Display (HMD) and camera, and estimating an 
accurate position and pose of fiducial markers (page 4, column 2, lines 4-11), which reads on 
the claimed markers for calibrating a camera to track the motion of the model plane and the 
claimed determining a pose of the moving model plane according to the markers in the video 
data and calibration results. Kato discloses a 3D image data model of a product in a pose 
corresponding to the pose of the whiteboard and ID cards (Figures 3 and 4), which reads on the 
claimed creating a three dimensional image data model of a product in a pose corresponding to 
the pose of the moving model plane. Kato discloses the six markers are used to find the board 
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orientation and align virtual images (page 6, column 1, line 6 - column 2, line 5), which reads on 
the claimed orientation of the markers determines the orientation of the three dimensional image 
data model. Kato discloses a transformation of the relationship between marker coordinates and 
camera coordinates (page 4, column 2, line 12 - page 6, column 1, line 3) and the results of the 
virtual images (Figures 3 and 4), which reads on the claimed determining an image 
correspondence between the camera calibration results associated with the moving model plane 
and the three dimensional image data model. Kato discloses the final result of the 3D virtual 
objects on the whiteboard and ID cards (Figures 3 and 4), which reads on the claimed rendering a 
3D model of the product based on the image correspondence and the claimed generating an 
augmented reality video by superimposing the rendered 3D model of the product on the moving 
model plane in the video data. 

Regarding independent claim 54, claim 54 is similar in scope as to claim 46, thus the 
rejection for claim 46 hereinabove is applicable to claim 54. 

Regarding dependent claims 52 and 60, Kato discloses a human manipulating an ID card 
(Figure 3), which reads on the claimed video data includes images of a human manipulating the 
pose of the model plane, wherein a rendering of the three-dimensional image model is 
manipulated according to the pose of the model plane. 

Regarding dependent claim 53, Kato discloses the system uses computer vision 
techniques and have optimized algorithms for fast, accurate real time registration and convenient 
optical see-through HMD calibration (page, 2, column 1, lines 18 - 26), which reads on the 
claimed wherein the augmented reality video is produced in real time. 
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Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claims 48 and 56 are rejected under 35 U.S.C. 103(a) as being unpatentable over Kato et 
al. ("Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing 
System", 20 October 1999) in view of Harrington (US 6,898,307). 

Regarding dependent claims 48 and 56, Kato did not expressly disclose scaling the three- 
dimensional image data model to the model plane according to the markers, though Kato does 
disclose simple transformation that could include scaling and also, transformations of the 
relationship between marker coordinates and camera coordinates (page 4, column 2, line 12 - 
page 6, column 1, line 3) and the sizes of the virtual screens on two ID cards being of different 
sizes and positions (Figure 3). Harrington discloses the translation, rotation, scaling and 
perspective operations can be applied to the image using standard image processing methods, or 
specialized textual mapping hardware may be employed for improving processing speed (column 
5, lines 26 - 35). It would have been obvious for one of ordinary skill in the art at the time of the 
invention to modify Kato's system to scale the three-dimensional virtual image according to the 
markers. One would be motivated to do so because simple known algebraic transformation in 3- 
D space helps improve processing speed and scaling the virtual images depending on the markers 
would prevent inaccurate and/or obscured virtual images. 
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Claims 61 and 62 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Harrington (US 6,898,307) in view of Rhoads et al. (US 7,050,603). 

Regarding independent claim 62, Harrington discloses a camera 12 that captures the 
normal field of view of the user of the headset 10 that is substantially what the user would see 
without the headset 10 (column 4, lines 4 - 24) and the analog signal from the camera 12 is 
converted to a digitized frame that can be stored in the computer 1 8 for analysis (column 4, lines 
25-38) and the computer 1 8 must interpret the captured video frames to locate a piece of paper 
within the camera view (column 4, lines 38 - 67 and column 5, lines 1 - 34) and the preselected 
movable real object comprises a reference panel such as a screen, tablet or piece of paper and the 
identifying includes recognizing a corner of the panel (claim 2), which reads on the claimed 
providing video data comprising images of a moving model plane having markers. It is inherent 
that a piece a paper, screen, or tablet is substantially rectangular, which reads on the claimed 
model plane is a substantially rectangular plane. Harrington discloses marker is a corner of the 
panel (claim 2) and a piece of paper can be identified by its white color against a darker 
background (column 4, line 55 - column 5, line 3), which reads on the claimed marker is a 
graphic disposed on an upper surface thereof Harrington further discloses the unprojection 
module 82 that projects three-dimension object onto a two-dimensional viewing plane, where a 
two dimensional image results from the three dimensional object (column 7, lines 28 - 53), 
which reads on the claimed providing a three-dimensional image data model of a product. 
Harrington did not expressly disclose the three-dimensional image data model is an animation. 
Rhoads discloses embedding watermark in an animated 3D graphical objects and the embedded 
object can be composited with a video stream to form a video program, such as a movie or 
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television program and graphical objects that link to information or electronic commerce 
transactions can be added to a video product, such as a movie, when its converted from one 
format to another (column 1 8, lines 5-18). It would have been obvious for one of ordinary skill 
in the art at the time of the invention to modify Harrington's system by projecting 3D animated 
objects. One would be motivated to do so because 3D animated objects are used in many 
commonly used applications such as games (column 18, line 63 - column 19, line 5) or 
interactive shopping (column 20, lines 41 - 48) that provide users a more convincing virtual 
environment. Harrington further discloses the unprojection step comprises the mathematical 
calculation comprising taking a two-dimensional image and determining what three-dimensional 
object could have created that two-dimensional image with respect to what the object can be and 
where it could have been positioned and the subject invention utilizes known dimensions with 
regard to the camera system and viewing plane, in combination with assumed characteristics of 
the piece of paper (column 7, lines 28 - 53), which reads on the claimed determining a pose of 
the moving model plane according to the markers in the video data and the claimed determining 
an image correspondence between the moving model plane having markers and the three- 
dimensional image data model. Harrington discloses the camera provide the computer with the 
same view that the user sees so that the synthetic image is displayed or superimpose in a manner 
for comfortable and convenient interaction between the real reference item within the field of 
view that the user may be holding and manipulating (column 4, lines 4 - 37), which reads on the 
claimed producing the augmented reality video by rendering the three-dimensional image data 
model of the product superimposed on the moving model plane in the video data, Harrington 
discloses the user could select as the application image a particular internet browser engine 
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which, based upon the located orientation of the piece of paper, would be transformed to exactly 
match the detected orientation of the piece of paper so that the displayed image provides the 
illusion of the web page actually appearing on the blank piece of paper being held by the user 
and once the position of the paper page is determined, information can be used to correctly 
distort the application's image so it overlays the paper in the eyes of the user (column 5, lines 4 - 
35) and the unprojection step comprises the mathematical calculation comprising taking a two- 
dimensional image and determining what three-dimensional object could have created that two- 
dimensional image with respect to what the object can be and where it could have been 
positioned and the subject invention utilizes known dimensions with regard to the camera system 
and viewing plane, in combination with assumed characteristics of the piece of paper (column 7, 
lines 28 - 53), which reads on the claimed rendering of the three-dimensional image data model 
has substantially the pose of the moving model plane. Harrington discloses video capture 
hardware that converts the analog signal from the camera into a digitized frame that can be 
stored in the computer and the hardware outpus a signal to the computer as a digitized and 
processible representation of what the user sees from the camera's field of view and the video 
generation hardware takes a bit map of pixel values from the computer and converts them into a 
television format that can be displayed on the headset and where the signal generated by the 
camera is different from the signal displayed on the headset by the addition of the computer- 
generated electronic image to thereby form a synthetic image as an augmented-reality display 
(column 4, lines 25 - 38), which reads on the claimed converting the augmented reality video 
into a streaming video format and the claimed streaming the augmented reality video having the 
streaming video format over a communications network to a computer for displaying the 
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augmented reality video. Harrington did not expressly discloses encoding hyperlink information 
into the augmented reality video and accessing information about the corresponding product by 
selecting the corresponding product in the rendered three-dimensional form from the augmented 
reality video. Rhoads discloses watermark may carry information or programmatic action or link 
to external information or an action, such as retrieval and output of information stored elsewhere 
in a database, website, etc (column 4, lines 31-39) and watermark can be embedded 
immediately into a video object layer after a graphical model is rendered to the video object layer 
(column 4, lines 40 - 49) and users selecting a video object, retrieving the linked information or 
actions for the selected object and rendering them on its user interface (column 16, lines 16 - 
38). It would have been obvious for one of ordinary skill in the art at the time of the invention to 
further modify Harrington's system by embedding hyperlink information into 3D rendered 
objects when selected. One would be motivated to do so because this gives users the 
convenience of retrieving extra information about the selected desired object from an external 
source upon request. 

Regarding independent claim 61, claim 61 is similar in scope as to claim 62, thus the 
rejection for claim 62 hereinabove is applicable to claims 61. 

Claims 50, 51, 58, and 59 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Kato et al. ("Marker Tracking and HMD Calibration for a Video-based Augmented Reality 
Conferencing System", 20 October 1999) in view of Rhoads et al. (US 7,050,603). 

Regarding dependent claims 50, 51, 58, and 59, Kato did not expressly discloses 
encoding hyperlink information into the augmented reality video and accessing information 
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about the corresponding product by selecting the corresponding product in the rendered three- 
dimensional form from the augmented reality video. Rhoads discloses watermark may carry 
information or programmatic action or link to external information or an action, such as retrieval 
and output of information stored elsewhere in a database, website, etc (column 4, lines 31 - 39) 
and watermark can be embedded immediately into a video object layer after a graphical model is 
rendered to the video object layer (column 4, lines 40 - 49) and users selecting a video object, 
retrieving the linked information or actions for the selected object and rendering them on its user 
interface (column 16, lines 16-38). It would have been obvious for one of ordinary skill in the 
art at the time of the invention to further modify Kato's system by embedding hyperlink 
information into 3D rendered objects when selected. One would be motivated to do so because 
this gives users the convenience of retrieving extra information about the selected desired object 
from an external source upon request. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffrey J. Chow whose telephone number is (571)272-8078. The 
examiner can normally be reached on Monday - Friday 10:00AM - 5:00PM (EST). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka Chauhan can be reached on (571)-272-7782. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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